Recurrent Neural Network Training with Dark Knowledge Transfer

Recurrent neural networks (RNNs), particularly long short-term memory (LSTM), have gained much attention in automatic speech recognition (ASR). Although some successful stories have been reported, training RNNs remains highly challenging, especially with limited training data. Recent research found that a well-trained model can be used as a teacher to train other child models, by using the predictions generated by the teacher model as supervision. This knowledge transfer learning has been employed to train simple neural nets with a complex one, so that the final performance can reach a level that is infeasible to obtain by regular training. In this paper, we employ the knowledge transfer learning approach to train RNNs (precisely LSTM) using a deep neural network (DNN) model as the teacher. This is different from most of the existing research on knowledge transfer learning, since the teacher (DNN) is assumed to be weaker than the child (RNN); however, our experiments on an ASR task showed that it works fairly well: without applying any tricks on the learning scheme, this approach can train RNNs successfully even with limited training data.Comment: ICASSP 201

Characterizing the stabilization size for semi-implicit Fourier-spectral method to phase field equations

Recent results in the literature provide computational evidence that stabilized semi-implicit time-stepping method can efficiently simulate phase field problems involving fourth-order nonlinear dif- fusion, with typical examples like the Cahn-Hilliard equation and the thin film type equation. The up-to-date theoretical explanation of the numerical stability relies on the assumption that the deriva- tive of the nonlinear potential function satisfies a Lipschitz type condition, which in a rigorous sense, implies the boundedness of the numerical solution. In this work we remove the Lipschitz assumption on the nonlinearity and prove unconditional energy stability for the stabilized semi-implicit time-stepping methods. It is shown that the size of stabilization term depends on the initial energy and the perturba- tion parameter but is independent of the time step. The corresponding error analysis is also established under minimal nonlinearity and regularity assumptions

Gradient bounds for a thin film epitaxy equation

We consider a gradient flow modeling the epitaxial growth of thin films with slope selection. The surface height profile satisfies a nonlinear diffusion equation with biharmonic dissipation. We establish optimal local and global wellposedness for initial data with critical regularity. To understand the mechanism of slope selection and the dependence on the dissipation coefficient, we exhibit several lower and upper bounds for the gradient of the solution in physical dimensions $d\le 3$

Experimental analysis of computer system dependability

This paper reviews an area which has evolved over the past 15 years: experimental analysis of computer system dependability. Methodologies and advances are discussed for three basic approaches used in the area: simulated fault injection, physical fault injection, and measurement-based analysis. The three approaches are suited, respectively, to dependability evaluation in the three phases of a system's life: design phase, prototype phase, and operational phase. Before the discussion of these phases, several statistical techniques used in the area are introduced. For each phase, a classification of research methods or study topics is outlined, followed by discussion of these methods or topics as well as representative studies. The statistical techniques introduced include the estimation of parameters and confidence intervals, probability distribution characterization, and several multivariate analysis methods. Importance sampling, a statistical technique used to accelerate Monte Carlo simulation, is also introduced. The discussion of simulated fault injection covers electrical-level, logic-level, and function-level fault injection methods as well as representative simulation environments such as FOCUS and DEPEND. The discussion of physical fault injection covers hardware, software, and radiation fault injection methods as well as several software and hybrid tools including FIAT, FERARI, HYBRID, and FINE. The discussion of measurement-based analysis covers measurement and data processing techniques, basic error characterization, dependency analysis, Markov reward modeling, software-dependability, and fault diagnosis. The discussion involves several important issues studies in the area, including fault models, fast simulation techniques, workload/failure dependency, correlated failures, and software fault tolerance